Photoelectric conversion apparatus, photoelectric conversion system, moving body

ABSTRACT

A photoelectric conversion apparatus includes a first substrate including a pixel array including a plurality of pixels, a second substrate layered on the first substrate and including an AD conversion portion including a plurality of AD conversion circuits configured to convert a signal output from the first substrate into a digital signal, wherein the second substrate further includes a plurality of signal processing units including a first signal processing unit and a second signal processing unit both configured to perform machine learning processing, wherein each of a plurality of sets includes a plurality of AD conversion circuits that differ between the plurality of sets, wherein the first signal processing unit is arranged to correspond to one of the plurality of sets, and wherein the second signal processing unit is arranged to correspond to another one of the plurality of sets.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a photoelectric conversion apparatus, aphotoelectric conversion system and a moving body.

Description of the Related Art

There is known a photoelectric conversion apparatus for convertingincident light into electric charges that has a layered structure with aplurality of substrates layered therein.

Japanese Patent Application Laid-Open Publication No. 2020-25263discusses a layered type light receiving sensor including a firstsubstrate and a second substrate layered therein. The first substrateincludes pixels, and the second substrate includes a signal processingcircuit (digital signal processor (DSP)). The signal processing circuitperforms processing based on a neural network calculation model.

SUMMARY OF THE INVENTION

A photoelectric conversion apparatus includes a first substrateincluding a pixel array including a plurality of pixels, a secondsubstrate layered on the first substrate and including an analog todigital (AD) conversion portion including a plurality of AD conversioncircuits configured to convert a signal output from the first substrateinto a digital signal, wherein the second substrate further includes aplurality of signal processing units including a first signal processingunit configured to perform machine learning processing and a secondsignal processing unit configured to perform machine learningprocessing, wherein each of a plurality of sets includes a plurality ofAD conversion circuits, and a plurality of AD conversion circuits of aset of the plurality of sets is different from a plurality of ADconversion circuits of another set of the plurality of sets, wherein thefirst signal processing unit is arranged to correspond to one of theplurality of sets, and wherein the second signal processing unit isarranged to correspond to another one of the plurality of sets.

Further features of the present invention will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams illustrating a structure of a photoelectricconversion apparatus.

FIG. 2 is a diagram illustrating a pixel structure.

FIG. 3 is a diagram illustrating a structure of a second substrateaccording to a first embodiment.

FIG. 4 is a diagram illustrating a structure of a second substrateaccording to a second embodiment.

FIG. 5 is a diagram illustrating a structure of the second substrateaccording to the second embodiment.

FIG. 6 is a diagram illustrating a structure of a second substrateaccording to a third embodiment.

FIG. 7 is a diagram illustrating an operation of the second substrateaccording to the third embodiment.

FIG. 8 is a diagram illustrating a structure of a second substrateaccording to a fourth embodiment.

FIG. 9 is a diagram illustrating a structure of a second substrateaccording to a fifth embodiment.

FIG. 10 is a diagram illustrating a structure of a second substrateaccording to a sixth embodiment.

FIG. 11 is a diagram illustrating an operation of the photoelectricconversion apparatus according to the sixth embodiment.

FIG. 12 is a diagram illustrating an operation of a photoelectricconversion apparatus according to a seventh embodiment.

FIG. 13 is a diagram illustrating an operation of the photoelectricconversion apparatus according to the seventh embodiment.

FIG. 14 is a diagram illustrating a structure of a second substrateaccording to the seventh embodiment.

FIG. 15 is a functional block diagram illustrating a photoelectricconversion system according to an eighth embodiment.

FIG. 16 is a functional block diagram illustrating a distance sensor.

FIG. 17 is a functional block diagram illustrating an endoscopic surgerysystem.

FIG. 18A is a diagram illustrating a photoelectric conversion system,and FIG. 18B is a diagram illustrating a moving body.

FIGS. 19A and 19B are schematic views illustrating smart glasses.

FIG. 20 is a functional block diagram illustrating a diagnosis system.

DESCRIPTION OF THE EMBODIMENTS

A processing signal processing circuit of a second substrate that isbased on a neural network calculation model has a high power consumptionand generates heat that increases proportionally to the powerconsumption. The heat generated by the second substrate is transmittedto pixel arrays of a first substrate. Consequently, signals output fromthe pixels contain more noise. Especially local heat generation causesuneven output in an image surface. This decreases image quality and alsomakes it difficult to perform image quality correction processing.

Further, as the function of processing based on a neural networkcalculation model has been advanced, a signal processing method in whicha plurality of processes is sequentially performed inhibits the signalprocessing speed from increasing.

The present disclosure relates to a technique advantageous to dispersionof heat generated by a second substrate and increase of the speed ofprocessing involving machine learning by the second substrate.

Various embodiments will be described below with reference to thedrawings.

In the following embodiments, mainly a photoelectric conversionapparatus will be described as an example of a photoelectric conversionapparatus. It should be noted, however, that the embodiments are notlimited to photoelectric conversion apparatuses and are also applicableto anything other than a photoelectric conversion apparatus. Examples ofother applications include a distance measurement apparatus (anapparatus for measuring a distance using focus detection ortime-of-flight (TOF) and a light measurement apparatus (an apparatus formeasuring an amount of incident light).

Each conductivity type of a transistor according to the below-describedembodiments is a mere example, and the conductivity types of thetransistors are not limited to those described below. The conductivitytypes according to the embodiments can be changed as needed, and in acase where the conductivity type of a transistor is changed, electricpotentials of a gate, a source, and a drain of the transistor are alsochanged as needed.

For example, in a case where the conductivity type of a transistor thatis operated as a switch is changed, an electric potential fed to a gateof the transistor is changed to a low level or a high level that isopposite to the level described in the embodiments. Further, eachconductivity type of a semiconductor region according to thebelow-described embodiments is a mere example, and the conductivitytypes of the semiconductor regions are not limited to those describedbelow. The conductivity types according to the embodiments can bechanged as needed, and in a case where the conductivity type of asemiconductor region is changed, an electric potential of thesemiconductor region is changed as needed.

FIG. 1A is a schematic diagram illustrating a layered-type photoelectricconversion apparatus according to a first embodiment. A first substrate100 and a second substrate 200 are semiconductor substrates, and thefirst substrate 100 includes a pixel array unit 110 including aplurality of unit pixels 101 arranged in a plurality of rows and aplurality of columns as illustrated in FIG. 1B. The plurality of unitpixels 101 can be arranged in a single row and a plurality of columns orcan be arranged in a single column and a plurality of rows. In aphotoelectric conversion apparatus for use in a digital camera, severaltens of millions of the unit pixels 101 are normally arranged.

The photoelectric conversion apparatus according to the presentembodiment is a backside-illuminated photoelectric conversion apparatusin which light enters from the first substrate 100 side. A signal line(not illustrated) is provided between this photoelectric conversionportion and a joint surface 300.

FIG. 2 is a circuit diagram illustrating circuits of two rows by twocolumns of the unit pixels 101 among the unit pixels 101 illustrated inFIG. 1 . Hereinafter, electric charges that photodiodes as aphotoelectric conversion portion accumulate are electrons. Alltransistors of the unit pixel 101 are N-type transistors. Alternatively,electric charges that photodiodes accumulate can be holes. In this case,the transistors of the unit pixels 101 can be P-type transistors. Inother words, the conductivity type defined below can be changed based onthe polarity of an electric charge as a signal.

Each unit pixel 101 includes a photodiode D1 serving as a photoelectricconversion portion, a transfer transistor M1, an electric chargeconversion portion C1, a reset transistor M3, an amplificationtransistor M4, and a selection transistor M5. The transfer transistor M1is on an electric path between the photodiode D1 and a node to which theelectric charge conversion portion C1, the reset transistor M3, and theamplification transistor M4 are connected. The electric chargeconversion portion C1 is also referred to as a floating diffusionportion (FD portion). A power supply voltage VDD is fed to the resettransistor M3 and the amplification transistor M4. The selectiontransistor M5 is on an electric path between the amplificationtransistor M4 and a column signal line 10. It can be said that theamplification transistor M4 is electrically connected to the verticaloutput line (column signal line) 10 via the selection transistor M5. Theelectric charge conversion portion C1 includes a floating diffusioncapacitance in the semiconductor substrate and a parasitic capacitanceon an electric path from the transfer transistor M1 to the amplificationtransistor M4 via the floating diffusion capacitance.

A signal RES, a signal Tx_A, and a signal SEL are signals fed from avertical scan circuit (not illustrated) via control lines illustrated inFIG. 2 . In FIG. 2 , the pixel row to which a signal is fed is specifiedat the end of the signal. For example, the signal RES(m) refers to thesignal RES that is fed to the unit pixels 101 of the mth row.

A current source (not illustrated) is connected to each of verticaloutput lines 10-1 and 10-2. In a case where the signal SEL(m) is changedto an active level, the selection transistors M5 of the unit pixels 101of the mth row are turned on. Consequently, an electric current is fedfrom the current source to the amplification transistors M4 of the unitpixels 101 of the mth row. In each unit pixel 101 of the mth row, thepower supply voltage VDD, the amplification transistor M4, and thecurrent source (not illustrated) connected to the vertical output line10-1 form a source follower circuit. Since the source follower circuitis formed, the amplification transistor M4 outputs a signal based on anelectric potential of the electric charge conversion portion C1 to thevertical output line 10-1 via the selection transistor M5.

Further, in a case where the signal SEL(m+1) is changed to an activelevel, the selection transistors M5 of the unit pixels 101 of the(m+1)th row are turned on. Consequently, an electric current is fed fromthe current source to the amplification transistor M4 of the (m+1)throw. In each unit pixel 101 of the (m+1)th row, the power supply voltageVDD, the amplification transistor M4, and the current source (notillustrated) connected to the vertical output line 10-2 form a sourcefollower circuit. Since the source follower circuit is formed, theamplification transistor M4 outputs a signal based on the electricpotential of the electric charge conversion portion C1 to the verticaloutput line 10-2 via the selection transistor M5.

As described above, the unit pixels 101 of the mth row and the unitpixels 101 of the (m+1)th row are connected to the different verticaloutput lines 10.

The structure illustrated in FIG. 2 is a mere example, and one verticaloutput line 10 or two or more vertical output lines 10 can be providedto a single column of the unit pixels 101. Further, the photoelectricconversion portion can be an avalanche photodiode or can be anythingthat performs photoelectric conversion.

The second substrate 200 includes a plurality of analog/digital (AD)conversion circuits 201 a to 201 h as illustrated in FIG. 3 . Theplurality of AD conversion circuits 201 a to 201 h converts analogsignals output from the unit pixels 101 into digital signals. The secondsubstrate 200 further includes a plurality of preprocessing units 202 ato 202 d. The plurality of preprocessing units 202 a to 202 d convertsdigital data output from the AD conversion circuits 201 a to 201 h intoimage data.

The second substrate 200 further includes a plurality of artificialintelligence (AI) processing units 203 a and 203 b. The plurality of AIprocessing units 203 a and 203 b each serve as a signal processing unitthat executes processing based on a neural network calculation model onimage data converted by the preprocessing units 202 a to 202 d. The AIprocessing units 203 a and 203 b include a memory unit storing a trainedmodel trained with a weight coefficient of a neural network.

The first substrate 100 and the second substrate 200 illustrated inFIGS. 1A, 1B, and 3 are joined together with the joint surface 300 toform the layered-type photoelectric conversion apparatus.

In FIGS. 1A, 1B, and 3 , the unit pixels 101, the AD conversion circuits201 a to 201 h, the preprocessing units 202 a to 202 d, and the AIprocessing units 203 a and 203 b are illustrated as elements of thephotoelectric conversion apparatus. Besides the foregoing elements, thefirst substrate 100 includes the control lines for controlling the unitpixels 101 and the vertical output lines 10 for transmitting signalsoutput from the unit pixels 101 as appropriate, as illustrated in FIG. 2. Further, the first substrate 100 or the second substrate 200 includesthe vertical scan circuit, a driving circuit such as a timing generator,and an output circuit for outputting image data as needed.

An analog signal output from each of the unit pixels 101 of the firstsubstrate 100 is input to a nearby AD conversion circuit among the ADconversion circuits 201 a to 201 h of the second substrate 200 based onthe location of the unit pixel 101 on the first substrate 100. Forexample, an analog signal output from the upper left pixel 101 viewedfrom the top in FIG. 1B is input to the AD conversion circuit 201 a atthe upper left of the second substrate 200. Similarly, an analog signaloutput from the lower right pixel 101 is input to the AD conversioncircuit 201 h at the lower right of the second substrate 200. Each ofthe AD conversion circuits 201 a to 201 h includes a plurality of ADconverters arranged in a plurality of rows and a plurality of columns.Each of the plurality of AD converters is arranged to correspond to oneof the vertical output lines 10-n illustrated in FIG. 2 . The ADconversion form of the AD converters is not particularly limited, andvarious forms of AD conversion such as slope AD conversion, ΔΣ ADconversion, and sequential comparison AD conversion can be applied.

Next, digital data, i.e., a digital signal output from each of the ADconversion circuits 201 a to 201 h, is input to a nearby preprocessingunit among the preprocessing units 202 a to 202 d of the secondsubstrate 200 based on the location of the AD conversion circuit on thesecond substrate 200. For example, digital data output from the ADconversion circuit 201 a or 201 b is input to the preprocessing units202 a. Similarly, digital data output from the AD conversion circuit 201g or 201 h is input to the preprocessing unit 202 d.

The preprocessing units 202 a to 202 d each perform signal processing ona digital signal output from the corresponding AD conversion circuits201. A process corresponding to part of image processing, such ascorrelated double sampling (CDS), offset removal, and amplificationprocessing, can be performed in the signal processing. For example, in acase where processing target image data is a color image, thepreprocessing unit 202 converts the format of the image data into aluma-blue-red (YUV) image data format or a red-green-blue (RGB) imagedata format. Further, the preprocessing unit 202 performs, for example,processing such as noise removal and white balance adjustment as neededon processing target image data. Furthermore, the preprocessing unit 202performs various types of signal processing (also referred to as“preprocessing”) on processing target image data as needed for the AIprocessing unit 203 to process the image data.

The AI processing units 203 a and 203 b perform processing based on aneural network calculation model on image data converted by a nearbypreprocessing unit among the preprocessing units 202 a to 202 d. Forexample, image data converted by the preprocessing unit 202 a or 202 cis processed by the AI processing unit 203 a, whereas image dataconverted by the preprocessing unit 202 b or 202 d is processed by theAI processing unit 203 b.

An AD conversion portion includes AD conversion circuits 201 a to 201 h.The AD conversion portion includes two AD conversion circuits as aplurality of sets. One set includes the AD conversion circuits 201 a and201 b, and another includes the AD conversion circuits 201 c and 201 d.One preprocessing unit 202 is provided for two AD conversion circuits ofone set. In other words, one set includes two AD conversion circuits andone preprocessing unit. Each of the AI processing units 203 a and 203 bis arranged to correspond to two sets.

The AI processing unit 203 a serving as a first signal processing unitis arranged to correspond to the AD conversion circuits 201 a and 201 bincluded in one set of the plurality of sets. Further, the AI processingunit 203 b serving as a second signal processing unit is arranged tocorrespond to the AD conversion circuits 201 c and 201 d included inanother set of the plurality of sets. Further, the plurality of sets isarranged in a plurality of rows and a plurality of columns. Theplurality of sets is arranged between the first and second signalprocessing units.

The AI processing units 203 a and 203 b are respectively located on theleft and right sides of the second substrate 200 as illustrated in FIG.3 . Processing based on a neural network calculation model is generallyhigh in power consumption and causes the AI processing unit 203 togenerate a great amount of heat. Heat generated by the second substrate200 is transmitted to the first substrate 100 via the joint surface 300and received by the unit pixels 101 to cause an increase in dark currentin the pixel array unit 110, and an uneven temperature causesnon-uniform (uneven) dark current. The uneven dark current generated inthe pixel array unit 110 causes uneven output of image data obtainedfrom the layered-type photoelectric conversion apparatus, so that notonly the image quality decreases but also the image quality correctionprocessing becomes difficult.

With the plurality of AI processing units 203 arranged as illustrated inFIG. 3 , local heat generation in the second substrate 200 is reduced.This reduces the non-uniformity of dark current and reduces unevenoutput of image data. Further, a plurality of AD conversion circuitsforms a single set, and an AI processing unit 203 is arranged tocorrespond to each of the plurality of sets. This enables parallelprocessing to increase the speed of machine learning processing.

Further, the AI processing units 203 a and 203 b are arranged tosandwich the AD conversion portion (AD conversion circuits 201 a to 201h) of the second substrate 200 so that heat generated by the AIprocessing unit 203 is suitably dispersed. This suitably reduces aneffect of heat generated at the second substrate 200 on the pixel arrayunit 110 of the first substrate 100.

While the AI processing units 203 a and 203 b are arranged near the leftand right sides of the second substrate 200 according to the presentembodiment, the arrangement is not limited to the above-describedarrangement, and the AI processing units 203 a and 203 b can be arrangednear upper and lower sides of the second substrate 200. Further, whilethe structure in which the first substrate 100 and the second substrate200 are layered is described as an example according to the presentembodiment, the structure is not limited to the above-describedstructure, and another semiconductor substrate can also be included. Forexample, a third substrate can be provided between the first substrate100 and the second substrate 200. The third substrate can include amemory element.

Further, in order to increase the effect of dispersion of heat generatedby the second substrate 200, the AI processing units 203 are desirablyarranged near two sides opposed to each other, three sides, or foursides of the second substrate 200.

Furthermore, the AI processing unit 203 is arranged in a region that isnot immediately below the pixel array unit 110 of the first substrate100. This makes it possible to minimize the effect of heat generated bythe AI processing unit 203 on the unit pixels 101.

Even in a case where the AI processing unit 203 is arranged immediatelybelow the pixel array unit 110, a decrease in image data quality isstill prevented by arranging the AI processing unit 203 in a region thatis not immediately below a light-shielded pixel region described belowin the pixel array unit 110.

Light-shielded pixels are pixels provided to detect an optical blacklevel (black level) and are shielded from light by a light shieldingfilm such as a metal. In a case where the amount of generated heat thatthe light-shielded pixels receive is small, the optical black level isnormally acquired, and an output value of each unit pixel other than thelight-shielded pixels that is changed by the generated heat can becorrected.

The light-shielded pixels may be arranged to surround the four sides ofthe pixel array unit 110 in some cases but may be also arranged alongtwo sides in L-shape. In this case, the AI processing units 203 can bearranged only near the two sides where the light-shielded pixels are notarranged and no AI processing unit 203 is arranged near the two sideswhere the light-shielded pixels are arranged on a plan view of the pixelarray unit 110 projected to the second substrate 200. In this case, theeffect of heat generated by the AI processing unit 203 on thelight-shielded pixels is reduced.

Another structure of the AI processing units 203 a and 203 b accordingto a second embodiment that is different from the structure according tothe first embodiment will be described below.

FIG. 4 illustrates an example of a structure of a second substrate 210of the photoelectric conversion apparatus according to the presentembodiment. The components other than the AI processing units 203 c and203 d are similar to those according to the first embodiment, so thatredundant descriptions thereof are omitted.

According to the present embodiment, the AI processing units 203 c and203 d have a similar structure to each other and are symmetricallyarranged along left and right edges of the second substrate 210,respectively.

FIG. 5 is an enlarged view of a portion enclosed by a dashed-lineillustrated in FIG. 4 . The AI processing unit 203 d includes n piecesof AI processing circuits 204 having the same function, and the AIprocessing circuits 204 (i.e., AI processing circuits 1 to n) areelectrically connected directly to the preprocessing units 202 b and 202d. According to the present embodiment, the preprocessing unit 202 b isconnected to the AI processing circuits 1, 2, 3, and 4, and thepreprocessing unit 202 d is directly connected to the AI processingcircuits 5 and n.

According to the present embodiment, the number of the plurality of AIprocessing circuits 204 of the plurality of AI processing units 203 isgreater than the number of the preprocessing circuits 202 of the secondsubstrate 200. This suitably reduces heat generated by machine learningprocessing.

According to the present embodiment, the signal processing units thatperform processing based on a neural network calculation model are moredispersed so that heat generated by the second substrate 210 can bedispersed. This reduces the effect of heat generated at the secondsubstrate 200 on the pixel array unit 110 of the first substrate 100.

As described above, the processing based on a neural network calculationmodel is performed in parallel using the plurality of AI processingcircuits 204 to increase the processing speed.

A modified example of the AI processing unit 203 according to the secondembodiment will be described below as a third embodiment. FIG. 6 is anenlarged view illustrating the portion enclosed by the dashed-lineaccording to the second embodiment illustrated in FIG. 4 . According tothe present embodiment, an AI processing unit 203 e illustrated in FIG.6 is provided as the AI processing unit 203 d illustrated in FIG. 4 .The AI processing unit 203 c illustrated in FIG. 4 includes a similarstructure to the structure of the AI processing unit 203 e illustratedin FIG. 6 .

The components other than the AI processing unit 203 e are similar tothose according to the first or second embodiment, so that redundantdescriptions thereof are omitted.

The AI processing unit 203 e includes n pieces of AI processing circuits205 having a fixed circuit structure configured to executestage-by-stage data processing, and each of the AI processing circuits205(1) to 205(n) is electrically connected in series.

In the present embodiment, execution of AI processing in three stageswill be described as an example. Image data converted by thepreprocessing unit 202 is passed to the AI processing circuits 205(1),205(2), and 205(3) in this order, and each of the AI processing circuits205(1), 205(2), and 205(3) performs processing based on a neural networkcalculation model.

Two-parallel processing of image data converted by the preprocessingunits 202 b and 202 d located on the upper and lower sides of the secondsubstrate 200 will be described as an example according to the presentembodiment will be described below. At this time, the AI processingcircuit 205(1) is electrically connected directly to the preprocessingunit 202 b, and the AI processing circuit 205(n) is electricallyconnected directly to the preprocessing unit 202 d.

FIG. 7 is a timing chart schematically illustrating calculationoperations based on a neural network calculation model that areperformed by the AI processing circuits 205 according to the presentembodiment. From time t1 to time t2, the AI processing circuit 205(1)performs processing based on a neural network calculation model on imagedata (hereinafter, referred to as “image data c”) converted by thepreprocessing unit 202 b. The image data c is based on digital dataoutput from the AD conversion circuit 201 c.

Next, from time t2 to time t3, the AI processing circuit 205(1) performsprocessing based on a neural network calculation model on image data(hereinafter, referred to as “image data d”) converted by thepreprocessing unit 202 b. The image data d is based on digital dataoutput from the AD conversion circuit 201 d.

The image data c is processed by the AI processing circuit 205(1) fromtime t1 to time t2. Further, the image data c is processed by another AIprocessing circuit 205, i.e., AI processing circuit 205(2), from time t2to time t3. The AI processing circuits 205(1) and (2) have respectiveneural network calculation models different from each other. Thus, theAI processing circuit 205(2) performs processing based on a neuralnetwork calculation model different from that used in the processingperformed by the AI processing circuit 205(1).

From time t3 to time t4, AI processing circuit 205(2) processes theimage data d based on a neural network calculation model different fromthat used in the processing performed by the AI processing circuit205(1). Further, the AI processing circuit 205(3) processes the imagedata c based on a neural network calculation model different from thatused in the processing performed by the AI processing circuit 205(2).

From time t4 to time t5, the AI processing circuit 205(3) processes theimage data d based on a neural network calculation model different fromthat used in the processing performed by the AI processing circuit205(2). Image data converted by the preprocessing unit 202 d based ondigital data output from the AD conversion circuit 201 g will bereferred to as “image data g”. Further, image data converted by thepreprocessing unit 202 d based on digital data output from the ADconversion circuit 201 h will be referred to as “image data h”. Each ofthe image data g and the image data h is sequentially processed based onthe different neural network calculation models by the AI processingcircuits 205(n−2), 205(n−1), and 205(n) from time t4 to time t5. Thisprocess is as illustrated in FIG. 7 .

As described above, the AI processing unit 203 of the photoelectricconversion apparatus according to the present embodiment has amulti-stage pipeline structure including three stages and performsprocessing based on a neural network calculation model using asequential processing method.

The arrangement of the AI processing circuits 205 according to thepresent embodiment is a mere example, and the AI processing circuits 205are desirably connected and arranged as suitable for the amount of heateach of the AI processing circuits 205 generates and the number ofprocessing stages. In FIG. 6 , the AI processing circuits 205 arearranged in series from the upper and lower edges of the secondsubstrate 200 toward the center of the second substrate 210 in the planview seen from the top view of the second substrate 200. The arrangementis not limited to the above-described example, and the AI processingcircuits 205 can be arranged in series from the center of the secondsubstrate 200 toward the upper and lower edges of the second substrate200. Further, the AI processing circuits 205 connected to thepreprocessing unit 202 b can be arranged by the upper edge of the secondsubstrate 200, and the AI processing circuits 205 connected to thepreprocessing unit 202 d can be located at the center of the secondsubstrate 200.

In this case, image data converted by the preprocessing unit 202 ispassed in the direction from the upper edge side toward the lower edgeside of the second substrate 200 when viewed from above.

According to the present embodiment, the processing units that performprocessing based on a neural network calculation model are moredispersed to disperse heat generated by the second substrate 200. Thus,the effect of heat generated at the second substrate 200 on the pixelarray unit 110 of the first substrate 100 is reduced.

Furthermore, the processing based on a neural network calculation modelis performed in parallel by the plurality of AI processing circuits 205to increase the processing speed.

A different arrangement of AD conversion circuits and AI processingunits according to a fourth embodiment will be described below.

FIG. 8 is a diagram illustrating a structure of a second substrate 400according to the present embodiment. In a photoelectric conversionapparatus according to the present embodiment, the second substrate 400includes one AD converter 401 for each unit pixel 101 of the firstsubstrate 100. With this structure, analog signals output from all theunit pixels 101 are simultaneously and collectively converted intodigital data by the AD converters 401.

Preprocessing and AI processing units 402 a to 402 d illustrated in FIG.8 convert digital data converted by the AD converters 401 into imagedata. Furthermore, the preprocessing and AI processing units 402 a to402 d perform processing based on a neural network calculation model onthe converted image data. The preprocessing and AI processing units 402a to 402 d in FIG. 8 are also referred to as circuit regions wherepreprocessing and AI processing are performed.

In FIG. 8 , pads 800 are provided on the four sides of the secondsubstrate 200. A signal (including power supply voltage) from a sourceoutside the photoelectric conversion apparatus is input to the pads 800,or the pads 800 output a signal to a destination outside thephotoelectric conversion apparatus. The plurality of preprocessing andAI processing units 402 a to 402 d is located in a region between anouter periphery portion where the pads 800 are provided on the foursides and the AD conversion portion (a region formed by the ADconverters 401 arranged in a plurality of rows and a plurality ofcolumns). While the pads 800 are located on all the four sides of thesecond substrate 400 in FIG. 8 , the pads 800 may be provided on twoopposite sides of the second substrate 200.

Digital data output from an AD converter 401 is input to one of thepreprocessing and AI processing units 402 a to 402 d based on thelocation of the AD converter 401 on the second substrate 400. Forexample, digital data output from AD converters 401 in a pixel region(a), digital data output from AD converters 401 in a pixel region (b),digital data output from AD converters 401 in a pixel region (c), anddigital data output from AD converters 401 in a pixel region (d) in FIG.8 are respectively input to the preprocessing and AI processing units402 a, 402 b, 402 c, and 402 d.

As described above, a plurality of regions including elements configuredto perform processing based on a neural network calculation model isarranged at substantially regular intervals. This makes it possible todisperse heat generated by the preprocessing and AI processing units 402a to 402 d in the second substrate 400. Thus, the effect of heatgenerated at the second substrate 400 on the pixel array unit 110 of thefirst substrate 100 is reduced.

Furthermore, parallel processing based on a neural network calculationmodel is performed by the plurality of preprocessing and AI processingunits 402 a to 402 d to increase the processing speed as in the secondembodiment.

Further, AI processing units according to the present embodiment mayhave a circuit structure configured to perform stage-by-stage dataprocessing as in the third embodiment. Specifically, the AI processingcircuits are electrically connected together in series to have amulti-stage pipeline structure and perform processing based on a neuralnetwork calculation model using a sequential processing method. In thiscase, the AI processing circuits in the preprocessing and AI processingunits 402 a to 402 d each have a circuit structure capable of performingstage-by-stage data processing, and the preprocessing and AI processingunits 402 a to 402 d are electrically connected together in series. Asto a method of the connection, for example, the preprocessing and AIprocessing units 402 a, 402 b, 402 c, and 402 d connected togetheraround the second substrate 400 can be employed, or only part of thepreprocessing and AI processing units 402 a to 402 d can be connected.The preprocessing and AI processing units 402 a and 402 b are connectedtogether, and then the preprocessing and AI processing units 402 c and402 d are connected together. Then, the preprocessing and AI processingunits 402 a and 402 b and the preprocessing and AI processing units 402c and 402 d perform sequential processing. The sequential processing bythe preprocessing and AI processing units 402 a and 402 b and thesequential processing by the preprocessing and AI processing units 402 cand 402 d can be performed simultaneously in parallel.

Furthermore, a selection switch can be provided to an input stage of thepreprocessing and AI processing units 402 a to 402 d so that theconfigurations of the sequential processing and the parallel processingare made variable.

In the first to fourth embodiments, examples where a plurality of AIprocessing units arranged to correspond to a plurality of sets performssignal processing involving machine learning processing on digital dataof the corresponding set are described above.

According to a fifth embodiment, different AI processing units performsignal processing on different frames.

FIG. 9 is a diagram illustrating a structure of the second substrate 200according to the present embodiment.

A preprocessing unit 900 a outputs the same data to both of AIprocessing units 901 a and 901 b.

Further, a preprocessing unit 900 b outputs the same data to both of theAI processing units 901 a and 901 b. In other words, the same data isinput to the AI processing units 901 a and 901 b from the plurality ofpreprocessing units, i.e., preprocessing units 900 a and 900 b. Variousparameters of the AI processing units 901 a and 901 b are adjusted bymachine learning, and the parameters of the AI processing units 901 aand 901 b are different from each other. Thus, even in a case where thesame data is input to the AI processing units 901 a and 901 b, the AIprocessing units 901 a and 901 b may output different output results.

The output results of the AI processing units 901 a and 901 b are inputto an overall processing unit 910. In a case where the output results ofthe AI processing units 901 a and 901 b are different from each other,the overall processing unit 910 performs one of the followingoperations.

(1) The overall processing unit 910 selects an output result with highreliability from the output results of the AI processing units 901 a and901 b and outputs the selected output result to a destination outsidethe photoelectric conversion apparatus.

(2) The overall processing unit 910 including a lookup table selects aresult corresponding to the combination of the output results of the AIprocessing units 901 a and 901 b from the lookup table and outputs theselected result.

(3) The overall processing unit 910 outputs both of the output resultsof the AI processing units 901 a and 901 b to a destination outside thephotoelectric conversion apparatus and further outputs reliabilityinformation.

The reliability determination regarding the operation (1) may beperformed by referring to a previous output result of the AI processingunit 901, or reliability levels of the AI processing units 901 a and 901b may be provided from a source outside the photoelectric conversionapparatus. Further, the AI processing units 901 a and 901 b are eachcaused to output reliability information about the output results, andthe output result with the reliability information that is higher thanthe other can be selected.

In the operation (3), the AI processing units 901 a and 901 b are eachcaused to output reliability information about the output results, andthe reliability information is output to a destination outside thephotoelectric conversion apparatus.

As described above, the plurality of AI processing units 901 of thephotoelectric conversion apparatus according to the present embodimentperforms signal processing involving machine learning processing on thesame data. This increases the accuracy of processing results output fromthe AI processing units 901.

Further, the photoelectric conversion apparatus according to the presentembodiment can give redundancy to the AI processing units 901.Specifically, there may be a case where a failure or a significantdecrease in signal accuracy occurs in one of the AI processing units 901a and 901 b. In this case, the operation of the one of the AI processingunits 901 a and 901 b is stopped or the output result of the one of theAI processing units 901 a and 901 b is ignored, and the output result ofthe other one of the AI processing units 901 a and 901 b is selected.Thus, even in a case where a failure or a decrease in signal accuracyoccurs in any one of the AI processing units 901 a and 901 b, theoperation of the photoelectric conversion apparatus can be continued.

Further, the inclusion of the plurality of AI processing units 901 a and901 b produces an advantage that local heat concentration is preventedas in the first embodiment. Further, since the plurality of AIprocessing units 901 a and 901 b performs signal processing, theprocessing speed is increased compared to a case where a single AIprocessing unit 901 performs signal processing involving machinelearning processing a plurality of times.

According to a sixth embodiment, part of a plurality of AI processingunits and another part of the plurality of AI processing unitsalternately perform an operation on frames. This increases the framerate.

The photoelectric conversion apparatus according to the presentembodiment can have a structure similar to that according to the fifthembodiment or can have a structure including more AI processing units asillustrated in FIG. 10 . The structure illustrated in FIG. 10 will bedescribed below.

The structure illustrated in FIG. 10 includes AI processing units 921 ato 921 d. Further, AD conversion circuits 201 a to 201 h each can outputdigital data selectively to one of the preprocessing units 900 a and 900b as illustrated with respect to the AD conversion circuit a. Further,the AD conversion circuits 201 a to 201 h can each have a structure tooutput digital data to both of the preprocessing units 900 a and 900 bin parallel.

FIG. 11 is a diagram illustrating operations performed by the AIprocessing units according to the present embodiment.

Image data of frames to be processed by the AD conversion circuits 201 ato 201 h which corresponds to the output of one screen is illustrated.

The AI processing unit 921 a starts processing image data of the nthframe (n is a natural number). Thereafter, while the AI processing unit921 a is processing the image data, the AI processing unit 921 b startsprocessing image data of the (n+1)th frame. Similarly, while the AIprocessing units 921 a and 921 b are processing the image data, the AIprocessing unit 921 c starts processing image data of the (n+3)th frame.Similarly, while the AI processing units 921 a, 921 b, and 921 c areprocessing the image data, the AI processing unit 921 d startsprocessing image data of the (n+4)th frame.

Thereafter, the AI processing unit 921 a finishes processing the imagedata and then starts processing image data of the (n+5)th frame. Theoperations are repeated similarly thereafter.

Since the preprocessing unit 900 a can output digital data selectivelyto the AI processing units 921 a or 921 b, the image data of theplurality of frames can be allocated to the plurality of AI processingunits 921 a and 921 b frame by frame. Further, in the configurationillustrated in FIG. 10 , the AD conversion circuits 201 a to 201 h areconfigured to output digital data selectively to one of thepreprocessing units 900 a and 900 b to facilitate allocation of digitaldata of a plurality of frames to the plurality of AI processing units921 a and 921 b.

The present embodiment is not limited to the configuration illustratedin FIG. 10 . For example, the AD conversion circuits 201 a to 201 h areconfigured to output digital data selectively to one of thepreprocessing units 900 a and 900 b. Besides this configuration, thepreprocessing units 900 a and 900 b can be combined into a singlepreprocessing unit 900, and the single preprocessing unit 900 canallocate data to the four AI processing units 921 a to 921 d. Further,the number of AI processing units 921 is not limited to four and can beany number greater than or equal to two. Further, the AI processingunits 921 a to 921 d can have a common trained model. This makes itpossible to obtain output results with equivalent reliability accuracyeven in a case where different AI processing units 921 performprocessing on different frames.

The plurality of AI processing units 921 a to 921 d can have a commontrained model in a method described below. First, each of the pluralityof AI processing units 921 a to 921 d independently performs machinelearning. The machine learning can be performed either using or withoutusing training data. After the AI processing units 921 a to 921 d finishmachine learning, a signal indicating that an expected output result isknown is input to the photoelectric conversion apparatus. A descriptionwill be given of an example where an expected output result is “asubject is a human face” and a human face is imaged by the photoelectricconversion apparatus.

Output results of the AI processing units 921 a to 921 d are input tothe overall processing unit 910. There may be a case where one or someof the AI processing units 921 a to 921 d output the output result “asubject is a human face” while another one of the AI processing units921 a to 921 d outputs an output result other than “a subject is a humanface”. In this case, the overall processing unit 910 increases thereliability of each AI processing unit 921 that outputs the correctoutput result (i.e., “a subject is a human face”) among the AIprocessing units 921 a to 921 d. The photoelectric conversion apparatusrepeats the operation of comparing the expected output result to anactual output result of each AI processing unit 921. As a result, theoverall processing unit 910 identifies the AI processing unit 921 thatis likely to output the correct output result among the AI processingunits 921 a to 921 d. The overall processing unit 910 applies thetrained model of the identified AI processing unit 921 to the other AIprocessing units 921. As a result, the plurality of AI processing units921 a to 921 d has the common trained model with high reliability.

A seventh embodiment will be described below. Mainly a difference fromthe sixth embodiment will be described below.

The photoelectric conversion apparatus according to the presentembodiment can have a structure similar to that according to the sixthembodiment.

According to the present embodiment, the overall processing unit 910outputs a processing result to a destination outside the photoelectricconversion apparatus based on the output results of the plurality offrames output from the plurality of AI processing units 921 a to 921 d.

FIG. 12 is a diagram illustrating operations performed by the AIprocessing units 921 a to 921 d illustrated in FIG. 10 according to thepresent embodiment. A difference from the operations illustrated in FIG.11 is that the overall processing unit 910 performs overalldetermination based on output results of the plurality of AI processingunits 921 a to 921 d and outputs the processing result to a destinationoutside the photoelectric conversion apparatus.

In the overall determination, for example, the most common output resultamong the output results of the plurality of AI processing units 921 ato 921 d is selected and output. In this case, the plurality of AIprocessing units 921 a to 921 d can have the same trained model as inthe sixth embodiment.

Further, the plurality of AI processing units 921 a to 921 d can havedifferent trained models from each other. In this form, after the AIprocessing units 921 a to 921 d finish machine learning, a signalindicating that the expected output result is known is input to thephotoelectric conversion apparatus. An example where the expected outputresult is “a subject is a human face” and a human face is imaged by thephotoelectric conversion apparatus will be described below. Outputresults of the AI processing units 921 a to 921 d are input to theoverall processing unit 910. There may be a case where one or some ofthe AI processing units 921 a to 921 d output the output result “asubject is a human face” while another one of the AI processing units921 a to 921 d outputs an output result other than “a subject is a humanface”. In this case, the overall processing unit 910 increases thereliability of each AI processing unit 921 that outputs the correctoutput result (i.e., “a subject is a human face”) among the AIprocessing units 921 a to 921 d. The photoelectric conversion apparatusrepeats the operation of comparing the expected output result to anactual output result of each AI processing unit 921. As a result, theoverall processing unit 910 determines the reliability of each of the AIprocessing units 921 a to 921 d. Then, the overall processing unit 910adds a reliability parameter to the output results of the plurality ofAI processing units 921 a to 921 d in the operations illustrated in FIG.12 and determines a processing result to be output to a destinationoutside the photoelectric conversion apparatus.

As described above, according to the present embodiment, the overalldetermination of the processing results of the plurality of AIprocessing units 921 a to 921 d is performed to obtain a processingresult with higher reliability.

An example where the plurality of AI processing units 921 a to 921 dprocesses image data of different frames from each other is described inthe present embodiment. In another example, the plurality of AIprocessing units 921 a to 921 d can process image data of the same frameas illustrated in FIG. 13 . In this case, the overall processing unit910 may perform overall determination and output a processing result toa destination outside the photoelectric conversion apparatus asdescribed in the present embodiment.

As described above in the embodiments, the AI processing units arearranged and configured to operate as described in the above-describedembodiments so that a decrease in accuracy of image data obtained fromthe photoelectric conversion apparatus is prevented and the operationspeed of the AI processing units increases.

Further, it is also effective to arrange the AI processing units 203 aand 203 b outside the arrangement area of the pixel array unit 110 in aplan view when viewed from the top surface side of the first and secondsubstrates as illustrated in FIG. 14 . In FIG. 14 , a projectionposition of the pixel array unit 110 of the first substrate on thesecond substrate when seen in a plan view is specified. The AIprocessing units 203 a and 203 b are arranged not to overlap the pixelarray unit 110 when seen in a plan view. This reduces the effect of heatgenerated in the AI processing units 203 a and 203 b on the pixel arrayunit 110. The preprocessing units 202 a to 202 d are also arranged notto overlap the pixel array unit 110 when seen in a plan view. Thisreduces the effect of heat generated in the preprocessing units 202 a to202 d on the pixel array unit 110.

Further, the operation processing speeds of the plurality of AIprocessing units 203 a and 203 b can be different from each other. Inthis case, one of the AI processing units 203 a and 203 b that has ahigher operation processing speed can be arranged farther from theposition of the pixel array unit 110 than the other one of the AIprocessing units 203 a and 203 b that has a lower operation processingspeed when seen in a plan view. In this case, since an AI processingunit having a higher operation processing speed generates more heat, aneffect of heat generated in the AI processing units 203 a and 203 b onthe pixel array unit 110 is reduced.

Further, while the AI processing units 203 a and 203 b are provided onthe second substrate according to the present specification, an AIprocessing unit can further be provided on the first substrate.

FIG. 15 is a block diagram illustrating a configuration of aphotoelectric conversion system 11200 according to an eighth embodiment.The photoelectric conversion system 11200 according to the presentembodiment includes a photoelectric conversion apparatus 11204. Any oneof the photoelectric conversion apparatuses according to theabove-described embodiments can be applied to the photoelectricconversion apparatus 11204. The photoelectric conversion system 11200can be used as, for example, an imaging system. Specific examples ofimaging systems include digital still cameras, digital camcorders,monitoring cameras, and network cameras. FIG. 15 illustrates a digitalstill camera as an example of the photoelectric conversion system 11200.

The photoelectric conversion system 11200 illustrated in FIG. 15includes the photoelectric conversion apparatus 11204 and a lens 11202.The lens 11202 forms an optical image of a subject on the photoelectricconversion apparatus 11204. The photoelectric conversion system 11200also includes a diaphragm 11203 and a barrier 11201. The diaphragm 11203varies the amount of light that passes through the lens 11202, and thebarrier 11201 protects the lens 11202. The lens 11202 and the diaphragm11203 constitutes an optical system that condenses light to thephotoelectric conversion apparatus 11204.

The photoelectric conversion system 11200 includes a signal processingunit 11205. The signal processing unit 11205 processes output signalsthat are output from the photoelectric conversion apparatus 11204. Thesignal processing unit 11205 performs signal processing such as varioustypes of correction and compression as needed on an input signal andoutputs the processed signal. The photoelectric conversion system 11200further includes a buffer memory unit 11206 and an external interface(external I/F) unit 11209. The buffer memory unit 11206 temporarilystores image data, and the external I/F unit 11209 is an interface forcommunication with an external computer. The photoelectric conversionsystem 11200 further includes a recording medium 11211 and a recordingmedium control interface unit (recording medium control I/F) 11210. Therecording medium 11211 is a semiconductor memory for recording andreading captured data, and the recording medium control I/F unit 11210is an interface for recording to and reading from the recording medium11211. The recording medium 11211 can be built in the photoelectricconversion system 11200 or can be attachable to and removable from thephotoelectric conversion system 11200. Further, communication from therecording medium control I/F unit 11210 to the recording medium 11211and communication from the external I/F unit 11209 can be performedwirelessly.

The photoelectric conversion system 11200 further includes an overallcontrol/calculation unit 11208 and a timing generation unit 11207. Theoverall control/calculation unit 11208 performs various types ofcalculation and controls the entire digital still camera. The timinggeneration unit 11207 outputs various timing signals to thephotoelectric conversion apparatus 11204 and the signal processing unit11205. The timing signals can be input from an external source, and thephotoelectric conversion system 11200 can be configured to include atleast the photoelectric conversion apparatus 11204 and the signalprocessing unit 11205 for processing output signals that are output fromthe photoelectric conversion apparatus 11204. The overallcontrol/calculation unit 11208 and the timing generation unit 11207 canbe configured to perform part of or entire control function of thephotoelectric conversion apparatus 11204.

The photoelectric conversion apparatus 11204 outputs image signals tothe signal processing unit 11205. The signal processing unit 11205performs predetermined signal processing on an image signal output fromthe photoelectric conversion apparatus 11204 and outputs image data.Further, the signal processing unit 11205 generates an image using animage signal. The signal processing unit 11205 can perform distancemeasurement calculation on a signal output from the photoelectricconversion apparatus 11204. The signal processing unit 11205 and/or thetiming generation unit 11207 can be mounted on the photoelectricconversion apparatus. Specifically, the signal processing unit 11205 andthe timing generation unit 11207 can be provided on a substrateincluding pixels arranged thereon or on another substrate. Using aphotoelectric conversion apparatus according to any one of theabove-described embodiments, an imaging system capable of acquiringimages with better quality is realized.

A ninth embodiment will be described below. FIG. 16 is a block diagramillustrating an example of a configuration of a distance image sensorthat is an electron device using one of the photoelectric conversionapparatuses according to the above-described embodiments.

As illustrated in FIG. 16 , a distance image sensor 12401 includes anoptical system 12407, a photoelectric conversion apparatus 12408, animage processing circuit 12404, a monitor 12405, and a memory 12406. Thedistance image sensor 12401 acquires a distance image based on adistance to a subject by receiving light (modulated light, pulse light)emitted from a light source device 12409 toward the subject andreflected by a surface of the subject.

The optical system 12407 includes a single lens or a plurality oflenses. The optical system 12407 guides image light (incident light)from the subject to the photoelectric conversion apparatus 12408 andforms an image on a light receiving surface (sensor portion) of thephotoelectric conversion apparatus 12408.

A photoelectric conversion apparatus according to any one of theabove-described embodiments is applied to the photoelectric conversionapparatus 12408, and a distance signal indicating a distance that isobtained based on a received light signal output from the photoelectricconversion apparatus 12408 is fed to the image processing circuit 12404.

The image processing circuit 12404 performs image processing to generatea distance image based on the distance signal fed from the photoelectricconversion apparatus 12408. The distance image (image data) obtained bythe image processing is fed to the monitor 12405 and displayed thereonor is fed to the memory 12406 and stored (recorded) therein.

Applying the photoelectric conversion apparatus described above to thedistance image sensor 12401 having the foregoing configuration improvespixel characteristics, so that, for example, a more accurate distanceimage is obtained.

A tenth embodiment will be described below. A technique according to thepresent disclosure (present technique) is applicable to variousproducts. For example, the technique according to the present disclosureis applicable to an endoscopic surgery system.

FIG. 17 is a diagram illustrating an example of a schematicconfiguration of an endoscopic surgery system to which the techniqueaccording to the present disclosure (present technique) is applicable.

FIG. 17 illustrates an operator (doctor) 13131 performing an operationon a patient 13132 on a patient bed 13133 using an endoscopic surgerysystem 13003. As illustrated in FIG. 17 , the endoscopic surgery system13003 includes an endoscope 13100, a surgical instrument 13110, and acart 13134 on which various devices for the endoscopic operation areplaced.

The endoscope 13100 includes a lens barrel 13101 and a camera head13102. A region of the lens barrel 13101 up to a predetermined lengthfrom a tip end of the lens barrel 13101 is inserted into a body cavityof the patient 13132. The camera head 13102 is connected to a base endof the lens barrel 13101. While the endoscope 13100 as a rigid endoscopeincluding the rigid lens barrel 13101 is illustrated as an example, theendoscope 13100 can be a flexible endoscope including a flexible lensbarrel.

The tip end of the lens barrel 13101 includes an opening in which anobjective lens is fitted. A light source device 13203 is connected tothe endoscope 13100, and light generated by the light source device13203 is guided to the tip end of the lens barrel 13101 by a light guideextended in the lens barrel 13101. The light is directed toward anobservation target in the body cavity of the patient 13132 through theobjective lens to illuminate the observation target. The endoscope 13100can be a forward-viewing endoscope, a forward-oblique viewing endoscopeor a side-viewing endoscope.

The camera head 13102 includes an optical system and a photoelectricconversion apparatus therein, and reflection light (observation light)from the observation target is condensed to the photoelectric conversionapparatus by the optical system. The photoelectric conversion apparatusphotoelectrically converts the observation light and generates anelectric signal corresponding to the observation light, i.e., an imagesignal corresponding to the observation image. Any one of thephotoelectric conversion apparatuses according to the above-describedembodiments can be used as the photoelectric conversion apparatus. Theimage signal is transmitted as raw data to a camera control unit (CCU)13135.

The CCU 13135 includes a central processing unit (CPU) and/or a graphicsprocessing unit (GPU) and comprehensively controls operations of theendoscope 13100 and a display device 13136. Furthermore, the CCU 13135receives the image signal from the camera head 13102 and performsvarious types of image processing such as development processing(demosaicing processing) on the image signal to display an image basedon the image signal.

The display device 13136 displays an image based on the image signal onwhich the CCU 13135 has performed the image processing, based on thecontrol by the CCU 13135.

The light source device 13203 includes a light source such as a lightemitting diode (LED) and supplies illumination light for use in imaginga surgical site to the endoscope 13100.

An input device 13137 is an input interface of the endoscopic surgerysystem 13003. A user can input various types of information andinstructions to the endoscopic surgery system 13003 via the input device13137.

A treatment tool control device 13138 controls the driving of an energytreatment tool 13112 for cauterization or incision of tissue, or sealingof blood vessels.

The light source device 13203 for feeding illumination light to theendoscope 13100 in imaging a surgical site can include, for example, awhite light source including a LED, a laser light source, or acombination thereof. In a case where the white light source includes acombination of red, green and blue (RGB) laser light sources, the outputintensity and timing of each color (each wavelength) is controlled withgreat accuracy, so that the white balance adjustment of a captured imagecan be performed by the light source device 13203. Further, in thiscase, laser light beams from the RGB laser light sources illuminate theobservation target by time division, and the driving of an image sensorof the camera head 13102 is controlled in synchronization with theillumination timings, whereby images that respectively correspond to RGBare captured by time division. With this method, a color image can beobtained without providing a color filter to the image sensor.

Further, the driving of the light source device 13203 can be controlledto change the output light intensity at predetermined time intervals.The driving of the image sensor of the camera head 13102 is controlledin synchronization with the light intensity change timings to acquireimages by time division, and the acquired images are combined togetherto generate a high dynamic range image without underexposure andoverexposure.

Further, the light source device 13203 can be configured to feed lightof a predetermined wavelength band for special light observation. Inspecial light observation, for example, the wavelength dependency oflight absorption in body tissue is used. Specifically, predeterminedtissue, such as blood vessels in a superficial portion of a mucousmembrane, is illuminated with light of a narrower band than illuminationlight (i.e., white light) in normal observation to capture ahigh-contrast image. Alternatively, in special light observation,fluorescence observation can be performed to obtain an image usingfluorescence generated by irradiation with excited light. Influorescence observation, body tissue is irradiated with excited lightto observe fluorescence from the body tissue, or a reagent such asindocyanine green (ICG) is locally injected into the body tissue and thebody tissue is irradiated with excited light corresponding to thefluorescence wavelength of the reagent to obtain a fluorescent image.The light source device 13203 can be configured to feed narrow-bandlight for the special light observation and/or excited light.

A photoelectric conversion system and a moving body according to aneleventh embodiment will be described below with reference to FIGS. 18Aand 18B. FIGS. 18A and 18B are schematic views illustrating an exampleof a configuration of the photoelectric conversion system and the movingbody according to the present embodiment. An example of an on-vehiclecamera as a photoelectric conversion system according to the presentembodiment will be described below.

FIGS. 18A and 18B illustrate an example of a vehicle system and aphotoelectric conversion system included therein and configured toperform imaging. A photoelectric conversion system 14301 includesphotoelectric conversion apparatuses 14302, image preprocessing units14315, an integrated circuit 14303, and optical systems 14314. Eachoptical system 14314 forms an optical image of a subject on thecorresponding photoelectric conversion apparatus 14302. Thephotoelectric conversion apparatus 14302 converts the optical image ofthe subject formed by the optical system 14314 into an electric signal.The photoelectric conversion apparatus 14302 is a photoelectricconversion apparatus according to one of the above-describedembodiments. The image preprocessing unit 14315 performs predeterminedsignal processing on signals output from the photoelectric conversionapparatus 14302. The function of the image preprocessing unit 14315 canbe included in the photoelectric conversion apparatus 14302. Thephotoelectric conversion system 14301 includes at least two sets of theoptical system 14314, the photoelectric conversion apparatus 14302, andthe image preprocessing unit 14315, and the output from the imagepreprocessing unit 14315 of each set is input to the integrated circuit14303.

The integrated circuit 14303 is an integrated circuit for use in animaging system. The integrated circuit 14303 includes an imageprocessing unit 14304 including a memory 14305 therein, an opticaldistance measurement unit 14306, a distance measurement calculation unit14307, an object identification unit 14308, and an abnormality detectionunit 14309. The image processing unit 14304 performs image processingsuch as development processing and defect correction on an output signalfrom the image preprocessing unit 14315. The memory 14305 primarilystores a captured image and also stores the position of a defect inimaging pixels. The optical distance measurement unit 14306 performsfocusing on a subject and distance measurement. The distance measurementcalculation unit 14307 calculates range information based on a pluralityof pieces of image data acquired by the plurality of photoelectricconversion apparatuses 14302. The object identification unit 14308identifies a subject such as a vehicle, a road, a traffic sign, or aperson. In a case where the abnormality detection unit 14309 detects anabnormality in the photoelectric conversion apparatus 14302, theabnormality detection unit 14309 notifies a main control unit 14313 ofthe abnormality.

The integrated circuit 14303 can be realized by dedicated hardware, asoftware module, or a combination thereof. Further, the integratedcircuit 14303 can be realized by a field-programmable gate array (FPGA),an application-specific integrated circuit (ASIC), or a combinationthereof.

The main control unit 14313 comprehensively controls operations of thephotoelectric conversion system 14301, a vehicle sensor 14310, and acontrol unit 14320. Alternatively, a method can be employed in which themain control unit 14313 is not included and the photoelectric conversionsystem 14301, the vehicle sensor 14310, and the control unit 14320 eachinclude a communication interface to transmit and receive controlsignals to and from one another via a communication network (e.g.,controller area network (CAN) standard).

The integrated circuit 14303 has a function of transmitting a controlsignal or a setting value to the photoelectric conversion apparatus14302 based on a control signal received from the main control unit14313 or a control unit of the integrated circuit 14303.

The photoelectric conversion system 14301 is connected to the vehiclesensor 14310 and detects a driving state of the vehicle, such as avehicle speed, yaw rate, and rudder angle, the external environment ofthe vehicle, and states of other vehicles and obstacles. The vehiclesensor 14310 is also a distance information acquisition unit thatacquires distance information about a distance to a target object.Further, the photoelectric conversion system 14301 is connected to adriver assist control unit 14311. The driver assist control unit 14311performs various types of driver assistance such as automatic steering,automatic cruising, and collision prevention function. Especially in thecollision determination function, a possibility or a presence/absence ofa collision with another vehicle or an obstacle is estimated based ondetection results of the photoelectric conversion system 14301 and thevehicle sensor 14310. With this collision determination function,control is performed to avoid a collision in a case where a collision isestimated, or a safety device is activated in a case where a collisionhas occurred.

Further, the photoelectric conversion system 14301 is also connected toa warning apparatus 14312. The warning apparatus 14312 issues a warningto a driver based on a determination result of the collisiondetermination unit. For example, in a case where the collisiondetermination unit determines that there is a high possibility of acollision, the main control unit 14313 performs vehicle control to avoida collision or reduce damage by applying a brake, releasing anaccelerator, or reducing engine output.

The warning apparatus 14312 issues a warning to a user by producing analarm such as a sound, displaying warning information on a display unitscreen of a car navigation system or a meter panel, or vibrating a seatbelt or steering.

According to the present embodiment, the photoelectric conversion system14301 captures images of an area near the vehicle, such as the front orrear of the vehicle. FIG. 18B illustrates an example of an arrangementof the photoelectric conversion system 14301 in a case where thephotoelectric conversion system 14301 captures images of the front ofthe vehicle.

The two photoelectric conversion apparatuses 14302 are arranged on thefront of a vehicle 14300. Specifically, a central line with respect to amovement direction or outer shape (e.g., vehicle width) of the vehicle14300 is defined as a symmetry axis, and the two photoelectricconversion apparatuses 14302 are arranged symmetrically with respect tothe symmetry axis. This form is desirable in acquiring distanceinformation about the distance between the vehicle 14300 and a targetsubject and in determining the possibility of a collision.

Further, the photoelectric conversion apparatuses 14302 are desirablyarranged to not obstruct the field of view of a driver in visuallychecking the circumstances outside the vehicle 14300. The warningapparatus 14312 is desirably arranged at a position within the driver'sfield of view.

Further, while an example where a control is performed to avoid acollision with another vehicle is described above in the presentembodiment, the present embodiment can be also applied to the control todrive the vehicle automatically following another vehicle and thecontrol to drive the vehicle automatically not to drift from the lane.Further, the photoelectric conversion system 14301 is applicable to notonly a vehicle such as an automobile but also any moving bodies (movingapparatuses) such as ships, aircrafts, or industrial robots.Furthermore, in addition to moving bodies, the photoelectric conversionsystem 14301 is also applicable to devices that widely use objectidentification, such as an intelligent transport system (ITS).

The photoelectric conversion apparatus according to the presentembodiment can further be configured to acquire various types ofinformation such as distance information.

A twelfth embodiment will be described below. FIGS. 19A and 19Billustrates glasses (smart glasses) 16600 according to an applicationexample. The glasses 16600 include a photoelectric conversion apparatus16602. The photoelectric conversion apparatus 16602 is a photoelectricconversion apparatus according to one of the above-describedembodiments. Further, a display device including a light emitting devicesuch as an organic LED or a LED can be provided on a rear surface sideof a lens 16601. A single photoelectric conversion apparatus 16602 or aplurality of photoelectric conversion apparatuses 16602 may be providedin the glasses 16600. Further, a plurality of types of photoelectricconversion apparatuses can be used in combination. The location of thephotoelectric conversion apparatus 16602 is not limited to that in FIG.19A.

The glasses 16600 further include a control device 16603. The controldevice 16603 functions as a power source that supplies power to thephotoelectric conversion apparatus 16602 and the display device.Further, the control device 16603 controls operations of thephotoelectric conversion apparatus 16602 and the display device. Thelens 16601 forms an optical system for condensing light to thephotoelectric conversion apparatus 16602.

FIG. 19B illustrates glasses (smart glasses) 16610 according to anotherapplication example.

The glasses 16610 include a control device 16612, and the control device16612 includes a photoelectric conversion apparatus corresponding to thephotoelectric conversion apparatus 16602 and the display device. A lens16611 forms an optical system for projecting light emitted from thephotoelectric conversion apparatus and the display device in the controldevice 16612, and an image is projected to the lens 16611. The controldevice 16612 functions as a power source that supplies power to thephotoelectric conversion apparatus and the display device. The controldevice 16612 also controls operations of the photoelectric conversionapparatus and the display device. The control device 16612 can alsoinclude a line-of-sight detection unit that detects the line of sight ofthe wearer. Infrared light can be used in the line-of-sight detection.An infrared light emitting unit emits infrared light to the eyeballs ofa user gazing at a displayed image. The infrared light emitted by theinfrared light emitting unit and thereafter reflected from the eyeballsis detected by an imaging unit including a light receiving element toobtain a captured image of the eyeballs. The inclusion of a reducingunit that reduces light from the infrared light emitting unit to thedisplay unit in a plan view prevents a decrease in image quality.

The line of sight of the user with respect to the displayed image isdetected from the captured image of the eyeballs that is acquired byinfrared imaging. A publicly-known method is applicable to theline-of-sight detection using a captured eyeball image. For example, aline-of-sight detection method based on a Purkinje image of reflectionsof irradiation light at the cornea can be used.

More specifically, line-of-sight detection processing based on the pupilcenter cornea reflection method is performed. A line-of-sight vectorindicating an eyeball orientation (rotation angle) is calculated basedon an image of a pupil center included in the captured eyeball image andthe Purkinje image using the pupil center cornea reflection method todetect the line of sight of the user.

The display device according to the present embodiment includes thephotoelectric conversion apparatus including the light receiving elementand controls an image displayed on the display device based on theline-of-sight information about the user from the photoelectricconversion apparatus.

Specifically, a first field-of-view region and a second field-of-viewregion of the display device are determined based on the line-of-sightinformation. The first field-of-view region is a region at which theuser is gazing, and the second field-of-view region is a region otherthan the first field-of-view region. The first field-of-view region andthe second field-of-view region can be determined by a control device ofthe display device, or the first field-of-view region and the secondfield-of-view region that are determined by an external control devicecan be received. In the display region of the display device, thedisplay resolution of the first field-of-view region can be controlledto be higher than the display resolution of the second field-of-viewregion. Specifically, the resolution of the second field-of-view regioncan be set lower than the resolution of the first field-of-view region.

Further, the display region includes a first display region and a seconddisplay region different from the first display region, and ahigh-priority region can be determined from the first display region andthe second display region based on the line-of-sight information. Thefirst field-of-view region and the second field-of-view region can bedetermined by the control device of the display device, or the firstfield-of-view region and the second field-of-view region that aredetermined by an external control device can be received. The resolutionof the high-priority region can be controlled to be higher than theresolution of the region other than the high-priority region. In otherwords, the resolution of the region that is relatively lower in prioritycan be set low.

AI can be used in determining the first field-of-view region or thehigh-priority region. AI can be a model trained to estimate an angle ofa line of sight from an image of eyeballs and a distance to a targetobject in the extension of the line of sight using the image of theeyeballs and a viewing direction of the eyeballs of the image astraining data. An AI program can be stored in the display device, thephotoelectric conversion apparatus, or an external apparatus. In a casewhere an external apparatus stores the AI program, the AI program istransmitted to the display device via communication.

In a case where display control is performed based on visual recognitionand detection, the present embodiment is suitably applicable to smartglasses further including a photoelectric conversion apparatusconfigured to capture external images. Smart glasses are capable ofdisplaying captured external information in real time.

A system according to a thirteenth embodiment will be described belowwith reference to FIG. 20 . The present embodiment is applicable to apathological diagnosis system with which a doctor diagnoses a lesion byobserving cells and tissues obtained from a patient and a diagnosisassistance system assisting the pathological diagnosis system. Thesystem according to the present embodiment can be used to diagnose alesion based on an acquired image or can assist with the diagnosis.

As illustrated in FIG. 20 , the system according to the presentembodiment includes one or more pathological systems 15510. The systemcan further include an analysis unit 15530 and a medical informationsystem 15540.

Each of the one or more pathological systems 15510 is a system that isused mainly by a pathologist and is installed in, for example,laboratories and hospitals. The pathological systems 15510 can beinstalled in different hospitals from one another and are each connectedto the analysis unit 15530 and the medical information system 15540 viavarious networks such as a wide area network and a local area network.

The pathological systems 15510 each include a microscope 15511, a server15512, and a display device 15513.

The microscope 15511 has an optical microscope function and captures animage of an observation target placed on a glass slide to obtain apathological image as a digital image. The observation target can be,for example, a tissue or cell obtained from a patient, such as a pieceof an organ, saliva, or blood.

The server 15512 stores the pathological image obtained by themicroscope 15511 in a storage unit (not illustrated). Further, in a casewhere a browsing request is received, the server 15512 searches for apathological image stored in a memory and displays a detectedpathological image on the display device 15513. A display control devicecan be provided between the server 15512 and the display device 15513.

In a case where an observation target is a solid object such as a pieceof an organ, the observation target can be, for example, a stained thinslice of the solid object. The thin slice can be prepared by, forexample, slicing a block cut from a specimen such as an organ. Further,in slicing a block, the block can be fixed with paraffin.

The microscope 15511 can include a low-resolution imaging unit forimaging at low resolution and a high-resolution imaging unit for imagingat high resolution. The low-resolution imaging unit and high-resolutionimaging unit can be different optical systems or the same opticalsystem. In a case where the low-resolution imaging unit andhigh-resolution imaging unit are the same optical system, the resolutionof the microscope 15511 can be changed based on an imaging target.

An observation target is placed on a glass slide, and the glass slidewith the observation target thereon is placed on a stage situated withinan angle of view of the microscope 15511. The microscope 15511 firstacquires an entire image within the angle of view using thelow-resolution imaging unit and then identifies a region of theobservation target from the acquired entire image. Then, the microscope15511 divides the region where the observation target is into aplurality of regions having a predetermined-size, and the dividedregions are sequentially imaged using the high-resolution imaging unitto obtain a high-resolution image of each divided region. To switch atarget divided region, the stage, the imaging optical system, or bothcan be moved. Further, each divided region can include a regionoverlapping an adjacent divided region in order to prevent any regionfrom being missed in imaging due to unintentional slipping of the glassslide. Further, the entire image can contain identification informationfor associating the entire image with the patient. The identificationinformation can be, for example, a character string or a Quick Response(QR) Code®.

The high-resolution images obtained by the microscope 15511 are input tothe server 15512. The server 15512 can divide each high-resolution imageinto smaller partial images. After generating the partial images, theserver 15512 performs combining processing of combining a predeterminednumber of adjacent partial images together to generate a single image onall the partial images. This combining processing can be repeated untila single partial image is eventually generated. By this processing, apartial image group having a pyramid structure with each hierarchicallayer consisting of one or more partial images is generated. In thispyramid structure, a partial image of a layer and a partial image ofanother layer are the same in the number of pixels but are different inresolution from each other. For example, in a case where two by twopartial images, i.e., four partial images, are combined together togenerate a single partial image of an upper layer, the resolution of thepartial image of the upper layer is one-half the resolution of thepartial images of a lower layer that are used in the combining.

The partial image group having the pyramid structure as described aboveis formed so that the level of detail of each observation target to bedisplayed on the display device can be changed based on the hierarchicallayer to which the display target tile image belongs. For example, in acase where a partial image of the lowermost layer is used, a narrowregion of the observation target is displayed in detail, and in caseswhere partial images of upper layers are used, wider regions of theobservation target are displayed roughly.

The generated partial image group of the pyramid structure can be storedin, for example, a memory. In a case where an acquisition request foracquiring a partial image containing identification information isreceived from another apparatus (e.g., the analysis unit 15530), theserver 15512 transmits a partial image corresponding to theidentification information to the other apparatus.

A partial image as a pathological image can be generated for eachimaging condition such as a focal length and staining condition. In acase where a partial image is generated for each imaging condition, aspecific pathological image and another pathological image thatcorresponds to an imaging condition different from a specific imagingcondition and is of the same region as the specific pathological imagecan be displayed next to each other. The specific imaging condition canbe designated by a viewer. In a case where the viewer designates aplurality of imaging conditions, pathological images that correspond tothe imaging conditions and are of the same region can be displayed nextto each other.

Further, the server 15512 can store the partial image group having thepyramid structure in a storage apparatus other than the server 15512,such as a cloud server. Further, the partial image generation processingdescribed above can partially or entirely be performed by the cloudserver. Use of the partial images as described above enables a user tofeel as though the user is observing an observation target whilechanging an observation magnification. In other words, the displaycontrol described above can play a role similar to a virtual microscope.A virtual observation magnification herein corresponds to a resolution.

The medical information system 15540 can be referred to as an electronichealth record system and stores patient identification information,patient disease information, inspection information and imageinformation used in diagnosis, diagnosis result, and diagnosis-relatedinformation such as prescription medication. For example, a pathologicalimage obtained by imaging an observation target of a patient is firststored via the server 15512 and thereafter displayed on the displaydevice 15513. A pathologist using the pathological system 15510 performspathological diagnosis based on the pathological image displayed on thedisplay device 15513. A result of the pathological diagnosis performedby the pathologist is stored in the medical information system 15540.

The analysis unit 15530 can analyze the pathological image. In theanalysis, a trained model generated by machine learning can be used. Theanalysis unit 15530 can derive a result of classification of a specificregion or a tissue identification result as an analysis result.Furthermore, the analysis unit 15530 can derive an identificationresult, such as cell information, quantity, position, and luminanceinformation, and scoring information thereon. The information obtainedby the analysis unit 15530 can be displayed as diagnosis assistanceinformation on the display device 15513 of the pathological system15510.

The analysis unit 15530 can be a server system consisting of one or moreservers (including a cloud server). Further, the analysis unit 15530 canbe embedded in, for example, the server 15512 in the pathological system15510. Specifically, various types of analysis of the pathologicalimages can be performed in the pathological system 15510.

The photoelectric conversion apparatuses according to theabove-described embodiments are suitably applicable to, for example, themicroscope 15511 among those configurations described above.Specifically, the photoelectric conversion apparatuses according to theabove-described embodiments are applicable to the low-resolution imagingunit and/or the high-resolution imaging unit in the microscope 15511.This makes it possible to reduce the size of the low-resolution imagingunit and/or the high-resolution imaging unit to reduce the size of themicroscope 15511. This makes it easy to transport the microscope 15511,so that system introduction and system rearrangement are facilitated.Furthermore, an application of a photoelectric conversion apparatusaccording to any of the above-described embodiments makes it possible toperform a series of processes from the pathological image acquisition tothe pathological image analysis partially or entirely on the fly in themicroscope 15511. This makes it possible to output diagnosis assistanceinformation more promptly and accurately.

The above-described configurations are applicable not only to diagnosisassistance systems but also to biological microscopes such as confocalmicroscopes, fluorescent microscopes, and video microscopes. Theobservation target can be biological samples such as cultured cells,fertilized eggs, or sperms, biomaterials such as cell sheets orthree-dimensional cell tissues, or living organisms such as zebrafish ormouse. Further, the observation target can be observed not only on aglass slide but also on a well plate or a petri dish.

Further, a moving image can be generated from still images of anobservation target that are obtained using a microscope. For example, amoving image can be generated from still images captured consecutivelyduring a predetermined period, or an image sequence can be generatedfrom still images captured at predetermined time intervals. With amoving image generated from still images as described above, a dynamicfeature of an observation target, such as a movement such as apulsation, elongation, or migration of a cancer cell, nerve cell,cardiac muscle tissue, or sperm, or a division process of cultured cellsor fertilized eggs can be analyzed using machine learning.

Other Embodiments

While various embodiments are described above, the present invention isnot limited to the embodiments, and various changes and modificationsare possible. Further, the embodiments are applicable to one another.Specifically, part of an embodiment can be replaced with part of anotherembodiment, or part of an embodiment can be added to part of anotherembodiment. Further, part of an embodiment can be deleted.

The scope of the disclosure of the present specification is not limitedto what is described in the present specification but encompasses allmatters that can be understood from the present specification and thedrawings attached to the present specification. Further, the scope ofthe disclosure of the present specification encompasses complementarysets of concepts disclosed in the present specification. Specifically,for example, in a case where the present specification includes thephrase “A is greater than B” but does not include the phrase “A is notgreater than B”, it is understood that the present specificationdiscloses the case where “A is not greater than B” because the inclusionof the phrase “A is greater than B” is based on the assumption that thecase where “A is not greater than B” is considered.

Dispersion of heat generated in a second substrate and an increasedspeed of processing involving machine learning performed in the secondsubstrate are realized.

While the present invention has been described with reference toembodiments, it is to be understood that the invention is not limited tothe disclosed embodiments, but is defined by the scope of the followingclaims.

This application claims the benefit of Japanese Patent Application No.2021-016446, filed Feb. 4, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A photoelectric conversion apparatus comprising:a first substrate including a pixel array including a plurality ofpixels, the pixel array including a light shielded region including aplurality of light shielded pixels; a second substrate layered on thefirst substrate and including an analog to digital (AD) conversionportion including a plurality of sets of AD conversion circuitsconfigured to convert a signal output from the first substrate into adigital signal, wherein the second substrate further includes aplurality of signal processing units including a first signal processingunit configured to perform machine learning processing and a secondsignal processing unit configured to perform machine learningprocessing, wherein a plurality of AD conversion circuits of a set ofthe plurality of sets is different from a plurality of AD conversioncircuits of another set of the plurality of sets, wherein the firstsignal processing unit is arranged to correspond to one of the pluralityof sets, wherein the second signal processing unit is arranged tocorrespond to another one of the plurality of sets, and wherein on aplan view from the pixel array to the second substrate, the first signalprocessing unit and the second signal processing unit are arranged inpositions which do not overlap the light shielded region.
 2. Thephotoelectric conversion apparatus according to claim 1, wherein theplurality of sets is arranged in a plurality of rows and a plurality ofcolumns, and wherein the plurality of sets is between the first and thesecond signal processing units.
 3. The photoelectric conversionapparatus according to claim 1, wherein the plurality of signalprocessing units further includes a third signal processing unit and afourth signal processing unit, and wherein the AD conversion portion isprovided in a region surrounded by the first, the second, the third, andthe fourth signal processing units.
 4. The photoelectric conversionapparatus according to claim 2, wherein the plurality of signalprocessing units further includes a third signal processing unit and afourth signal processing unit, and wherein the AD conversion portion isprovided in a region surrounded by the first, the second, the third, andthe fourth signal processing units.
 5. The photoelectric conversionapparatus according to claim 3, wherein the first, the second, thethird, and the fourth signal processing units are arranged along anouter periphery of the second substrate.
 6. The photoelectric conversionapparatus according to claim 4, wherein the first, the second, thethird, and the fourth signal processing units are arranged along anouter periphery of the second substrate.
 7. The photoelectric conversionapparatus according to claim 5, wherein a plurality of pads to which asignal from a source outside the photoelectric conversion apparatus isinput or from which a signal is output to a destination outside thephotoelectric conversion apparatus is arranged along the outer peripheryof the second substrate, and wherein two or more of the first, thesecond, the third, and the fourth signal processing units are arrangedbetween the plurality of pads and the AD conversion portion.
 8. Thephotoelectric conversion apparatus according to claim 6, wherein aplurality of pads to which a signal from a source outside thephotoelectric conversion apparatus is input or from which a signal isoutput to a destination outside the photoelectric conversion apparatusis arranged along the outer periphery of the second substrate, andwherein two or more of the first, the second, the third, and the fourthsignal processing units are arranged between the plurality of pads andthe AD conversion portion.
 9. The photoelectric conversion apparatusaccording to claim 7, wherein all the plurality of signal processingunits are arranged between the plurality of pads and the AD conversionportion.
 10. The photoelectric conversion apparatus according to claim8, wherein all the plurality of signal processing units are arrangedbetween the plurality of pads and the AD conversion portion.
 11. Thephotoelectric conversion apparatus according to claim 1, wherein thefirst and the second signal processing units are different in signalprocessing speed from each other.
 12. The photoelectric conversionapparatus according to claim 2, wherein the first and the second signalprocessing units are different in signal processing speed from eachother.
 13. The photoelectric conversion apparatus according to claim 3,wherein the first and the second signal processing units are differentin signal processing speed from each other.
 14. The photoelectricconversion apparatus according to claim 5, wherein the first and thesecond signal processing units are different in signal processing speedfrom each other.
 15. The photoelectric conversion apparatus according toclaim 7, wherein the first and the second signal processing units aredifferent in signal processing speed from each other.
 16. Thephotoelectric conversion apparatus according to claim 1, wherein each ofthe plurality of sets includes a preprocessing circuit to which theplurality of AD conversion circuits of the set inputs the digitalsignal, and wherein each of the plurality of signal processing unitsincludes a plurality of signal processing circuits each configured toperform the machine learning, and a processing result of thepreprocessing circuit is input to the plurality of signal processingcircuits of the first signal processing unit.
 17. The photoelectricconversion apparatus according to claim 2, wherein each of the pluralityof sets includes a preprocessing circuit to which the plurality of ADconversion circuits of the set inputs the digital signal, and whereineach of the plurality of signal processing units includes a plurality ofsignal processing circuits each configured to perform the machinelearning, and a processing result of the preprocessing circuit is inputto the plurality of signal processing circuits of the first signalprocessing unit.
 18. The photoelectric conversion apparatus according toclaim 3, wherein each of the plurality of sets includes a preprocessingcircuit to which the plurality of AD conversion circuits of the setinputs the digital signal, and wherein each of the plurality of signalprocessing units includes a plurality of signal processing circuits eachconfigured to perform the machine learning, and a processing result ofthe preprocessing circuit is input to the plurality of signal processingcircuits of the first signal processing unit.
 19. The photoelectricconversion apparatus according to claim 16, wherein a number of theplurality of signal processing circuits of the plurality of signalprocessing units is greater than a number of the preprocessing circuitsprovided on the second substrate.
 20. A photoelectric conversion systemcomprising: a first substrate including a pixel array including aplurality of pixels; and a second substrate layered on the firstsubstrate and including an AD conversion portion configured to convert asignal output from the first substrate into a digital signal, whereinthe second substrate includes: the photoelectric conversion apparatusaccording to claim 1; and a signal processing unit configured togenerate an image using a signal output from the photoelectricconversion apparatus.
 21. A moving body including the photoelectricconversion apparatus according to claim 1, the moving body comprising: acontrol unit configured to control a movement of the moving body using asignal output from the photoelectric conversion apparatus.
 22. Thephotoelectric conversion apparatus according to claim 1, wherein each ofthe plurality of signal processing unit is for performing machinelearning processing, and wherein on the plan view, none of the pluralityof signal processing units overlaps the light shielded region.
 23. Thephotoelectric conversion apparatus according to claim 1, wherein thelight shielded region is arranged along two sides of the pixel array.24. The photoelectric conversion apparatus according to claim 23,wherein the light shielded region is in L shape.
 25. The photoelectricconversion apparatus according to claim 1, wherein the light shieldedregion is arranged along four sides of the pixel array.